Finding the WRITE Stuff: Automatic Identification of Discourse Structure in Student Essays

automated feedback that helps them revise their work and ultimately improve their writing skills. These applications also address educational researchers’ interest in individualized instruction. Specifically, feedback that refers explicitly to students’own writing is more effective than general feedback.3 Our discourse analysis software, which is embedded in Criterion (www.etstechnologies.com), an online essay evaluation application, uses machine learning to identify discourse elements in student essays. The system makes decisions that exemplify how teachers perform this task. For instance, when grading student essays, teachers comment on the discourse structure. Teachers might explicitly state that the essay lacks a thesis statement or that an essay’s single main idea has insufficient support. Training the systems to model this behavior requires human judges to annotate a data sample of student essays. The annotation schema reflects the highly structured discourse of genres such as persuasive writing. Our discourse analysis system uses a voting algorithm that takes into account the discourse labeling decisions of three independent systems. The three systems employ natural language processing methods to extract essay-based features that help predict the discourse labels. They also use machine learning to classify the sentences in an essay as particular discourse elements. Our tool automatically labels discourse elements in student essays written on any topic and across writing genres.